Labelled Message Routing

Overview

Labelled message routing allows you to isolate and independently scale different data processing flows within your EnergyWorx environment. By assigning labels to your data flows, you can:

• Guarantee resources for critical or high-priority data processing
• Prevent interference between different data flows (e.g., switching processes not impacting vee processes)
• Scale independently based on the specific needs of each data flow
• Optimize costs by allocating resources only where needed

Each label creates dedicated infrastructure that processes messages separately from other flows.

How to Use Labelled Routing

Step 1: Define Available Labels

Labels are configured on Process Pools. Process pools separate data processing into different compute environments, allowing namespaces that share a process pool to share processing resources.

To add or modify labels on a process pool:

1. Contact the service desk to request the desired labels on your process pool
2. A platform deployment will be scheduled to apply the changes

Example: A process pool with two labels

• high-priority
• batch-processing

info

Adding labels to a process pool requires a platform deployment. Contact the service desk to request label changes — they will coordinate the deployment.

Step 2: Route Data Flows to Labels

Configure your Transformation Configuration to use a label:

1. Open or create a Transformation Configuration
2. Add a flow property:
   • Key: crunch-topic-trigger-label
   • Value: The label name (must match a process pool label exactly)
3. Save the configuration

All data processed through this transformation configuration will now use the dedicated infrastructure for that label.

warning

The label value must exactly match a label defined on the process pool, or messages will fail to route correctly.

What Happens Behind the Scenes

When labels are added to a process pool, the platform automatically creates:

Dedicated Infrastructure per Label

Each label gets its own set of processing resources:

• Message Queues: Separate queues for each processing stage
• Processing Pipelines: Independent deployments that process only messages with that label
• Autoscaling: Each labelled pipeline scales independently based on its workload

Example: If process pool 1 has two labels (high-priority and batch-processing), you'll have:

• Default (unlabelled) infrastructure: Handles flows without labels
• high-priority infrastructure: Handles flows marked with this label
• batch-processing infrastructure: Handles flows marked with this label

Independent Scaling

Each labelled pipeline:

• Monitors its own message backlog: Only scales based on messages in its dedicated queue
• Scales automatically: More messages = more processing capacity
• Scales independently: High load on one label doesn't affect others
• Has resource limits: Maximum capacity can be configured per label

This means:

• Critical data flows can always get the resources they need
• Non-critical flows can scale down to save costs when idle
• Different types of data can have different processing capacities

Advanced: Namespace-Level Override

Administrators can disable labelled routing for an entire namespace if needed:

1. Add a namespace property:
   • Key: disable-labelled-messages
   • Value: true
2. Save the namespace

Effect: All data flows will use the default (unlabelled) infrastructure, regardless of transformation configuration settings. This is useful for:

• Troubleshooting routing issues
• Temporarily consolidating processing
• Emergency fallback scenarios

Cost and Resource Considerations

Infrastructure Multiplication

Each label creates dedicated infrastructure for processing. This means:

For each label you add:

• Additional message queues (topics and subscriptions)
• Additional processing pipelines (that can scale to multiple pods)
• Additional autoscaling configurations

Example: If you add 3 labels to a process pool:

• You'll have 4 sets of infrastructure total (default + 3 labels)
• Each set can independently scale from 1 to many processing pods
• Even when idle, there are baseline infrastructure costs

Cost Impact

Estimate per label: ~$120-150/month (varies by usage)

• Base infrastructure: Message queues and configurations
• Processing capacity: Scales with workload (more messages = more cost)
• Idle periods: Some costs even when not processing messages

When labels are cost-effective:

• High-volume data flows that need guaranteed resources
• Critical data that can't afford delays from other processing
• Distinct SLA requirements for different data types

When labels may not be worth it:

• Low-volume data flows (baseline costs may exceed benefits)
• Similar processing requirements across all data
• Budget-constrained environments with uniform data priorities

Use Cases

When to Use Labelled Message Routing

1. Priority-Based Processing

   • High-priority vs. low-priority data flows
   • Separate SLA requirements for different customers
   • Guaranteed resources for critical data

2. Workload Isolation

   • Isolate experimental or unstable transformations
   • Prevent noisy neighbors from affecting critical flows
   • Debug-specific data flows without impacting production

3. Independent Scaling

   • Different throughput requirements for different data types
   • Cost optimization by scaling only what's needed
   • Peak-time handling for specific data sources

4. Resource Allocation

   • Dedicated compute resources for specific customers
   • Compliance requirements for data separation
   • Multi-tenancy within a single namespace

When NOT to Use Labels

1. Few Messages

   • Low-volume flows don't benefit from separate infrastructure
   • Overhead of additional resources outweighs benefits

2. Uniform Processing Requirements

   • All data has similar priority and SLA
   • No need for isolation or independent scaling

3. Cost Sensitivity

   • Additional infrastructure increases costs
   • Each label adds fixed costs (even when idle)

First-Time Setup

Important - Initial Deployment

The first time labelled routing is enabled in an environment, there will be infrastructure changes that require:

• Planned downtime: Data processing must be stopped
• Empty queues: All pending messages must be processed or drained
• Coordination with platform administrators: This is a significant deployment

Contact your platform administrator well in advance to plan this deployment.

Adding New Labels

1. Contact the service desk to request the new label on your process pool
2. A platform deployment will be scheduled to create the infrastructure
3. After deployment, update transformation configurations to use the new label
4. Data will start routing to the new labelled infrastructure

caution

Wait for infrastructure deployment to complete before updating transformation configurations, or messages may fail to route.

Removing Labels

1. Update all transformation configurations to remove references to the label
2. Verify no data is still being processed with that label
3. Contact the service desk to request removal of the label from the process pool
4. A platform deployment will be scheduled to remove the infrastructure

danger

Removing a label while transformation configurations still use it will cause data processing failures. Always update configurations first!

Monitoring Your Labelled Flows

What we Watch

For each label you create, we monitor:

1. Processing Backlog

   • Are messages piling up faster than they can be processed?
   • May indicate need for more processing capacity

2. Processing Speed

   • How long does it take to process messages?
   • Compare against your SLA requirements

3. Error Rates

   • Are messages failing to process?
   • Check dead letter queues for failed messages

4. Resource Usage

   • How many processing pods are running?
   • Are you hitting capacity limits?

Where to Look

Your platform administrator can help you with the following information:

• Processing metrics dashboards
• Message queue statistics
• Error logs and dead letter queues
• Resource utilization reports

Questions to ask:

• "How is my {label-name} flow performing?"
• "Are we hitting capacity limits for {label-name}?"
• "Can we increase the maximum processing capacity for this label?"

Best Practices

Label Naming

Choose clear, descriptive names:

• Good: high-priority, customer-abc, batch-processing, real-time
• Avoid: label1, test, tmp, special characters

Tips:

• Use lowercase with hyphens
• Make it obvious what the label is for
• Keep names short but meaningful
• Be consistent across your environment

When to Use Labels

Good use cases:

• Priority separation: Critical vs. non-critical data
• Customer isolation: Premium customers get guaranteed resources
• SLA requirements: Different processing speed requirements
• Data type separation: Real-time vs. batch processing
• Compliance: Sensitive data needs isolated processing

When NOT to use labels:

• Low-volume data flows (overhead isn't worth it)
• Similar processing requirements for all data
• Budget constraints (labels add infrastructure costs)
• Testing or temporary needs (use separate environments instead)

Planning Your Labels

Before adding labels, consider:

1. Volume: How much data will use this label?

   • High volume = labels more cost-effective
   • Low volume = may not justify overhead

2. Requirements: What makes this data special?

   • Does it need faster processing?
   • Does it need guaranteed resources?
   • Does it have different SLAs?

3. Budget: Can you afford the additional infrastructure?

   • Each label ~$120-150/month base cost
   • Additional costs scale with usage

4. Complexity: Can you manage multiple flows?

   • More labels = more monitoring
   • More labels = more configuration
   • More labels = more troubleshooting

Start Small

Recommended approach:

1. Start with 1-2 labels for clear use cases
2. Monitor performance and costs
3. Add more labels only when clearly beneficial
4. Regularly review if labels are still needed

Documentation

For each label, document:

• Purpose: Why does this label exist?
• Data flows: Which transformation configurations use it?
• Requirements: What SLAs or guarantees does it provide?
• Owners: Who is responsible for data using this label?
• Review date: When should we re-evaluate if it's still needed?

Troubleshooting

Messages Aren't Routing to My Label

Check these things:

1. Label name matches exactly

   • Process pool label: high-priority
   • Transformation config property: crunch-topic-trigger-label = high-priority
   • They must match exactly (case-sensitive, no extra spaces)

2. Infrastructure deployed

   • Has a platform deployment happened since adding the label?
   • Ask your administrator if infrastructure exists for this label

3. Namespace override

   • Check if namespace has disable-labelled-messages = true
   • This would force all flows to use default routing

Still not working? Contact the service desk with:

• Process pool ID
• Label name
• Transformation configuration ID
• Error messages or symptoms

Processing is Slow or Backed Up

Possible causes:

1. Not enough processing capacity

   • Too many messages for current pipeline capacity
   • Solution: Ask administrator to increase maximum pods

2. Resource limits hit

   • Pipeline at maximum configured capacity
   • Solution: Request higher limits for this label

3. Messages going to wrong label

   • Check transformation configuration
   • Verify label names match exactly

To report: Tell your administrator:

• Which label is affected
• Current backlog size (if known)
• Expected vs. actual processing speed

Costs Higher Than Expected

Review with your administrator:

1. Are all labels still needed?

   • Audit which labels are actively used
   • Remove unused labels to save costs

2. Can labels be consolidated?

   • Maybe 3 labels could become 2
   • Combine similar use cases

3. Is scaling configured optimally?

   • Too much minimum capacity = wasted resources
   • Too aggressive scaling = unnecessary costs

Summary

Labelled message routing allows you to create dedicated, isolated processing infrastructure for different types of data flows. This enables:

• Guaranteed resources for critical data
• Independent scaling for different workloads
• Workload isolation to prevent interference

Key points to remember:

• Labels are defined on process pools (request through service desk)
• Transformation configurations reference those labels
• Each label creates separate infrastructure (costs ~$120-150/month)
• Labels scale independently based on their workload
• Use labels strategically for clear business value
• Changes require a platform deployment to take effect

For more details:

• Contact your platform administrator for label setup and management
• See Transformation Configuration for configuring data flows
• See Process Monitoring for monitoring your labelled flows